Optimised database registration logic in CLEM workflow#787
Merged
Conversation
… reference for when updating grid squares
tieneupin
added
bug
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #787 +/- ##
==========================================
+ Coverage 50.72% 50.84% +0.12%
==========================================
Files 96 96
Lines 10059 10082 +23
Branches 1322 1337 +15
==========================================
+ Hits 5102 5126 +24
+ Misses 4690 4687 -3
- Partials 267 269 +2 🚀 New features to boost your workflow:
|
…te a new ImagingSite entry, but should fail loudly if no entry was found; move the 'commit()' command out of the for loop
tieneupin
changed the title
tieneupin
changed the title
…entry; modified the subsequent helper functions to use the returned ImagingSite entry instead of performing new queries
stephen-riggs
approved these changes
Apr 30, 2026
Contributor
stephen-riggs
left a comment
stephen-riggs
left a comment
There was a problem hiding this comment.
This looks cleaner than before, thanks for making the discussed changes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes a couple of issues with the database registration logic in the CLEM workflow:
Handle the possibility of there being multiple atlas images associated with a grid
We occasionally run into situations where more than one image set for a grid meets the criteria to be registered as an atlas, which breaks the workflow because the SQL query expects a single matching database entry or no result.
This can be resolved by returning all matching atlas entries, sorted by insertion order. The latest one is used to perform the ISPyB updates. Because the
GridSquareandDataCollectionGroupentries are re-updated for every new image set received, all entries for a given sample grid will be re-updated correctly with reference to the last atlas registered.Accidental insert when updating
ImagingSiteentries after registeringGridSquareentriesThere was a race condition between when the
ImagingSiteentries from_register_clem_imaging_siteare committed and when they are queried in_register_dcg_and_atlas. This lead to the creation of duplicateImagingSiteentries in the database. This can be fixed by modifying the subsequent helper functions to use the registeredImagingSiteentry from_register_clem_imaging_siteinstead of querying it from the database.Less aggressive commits
In
register_grid_square,db.commit()was previously run multiple times within a singleforloop. By moving it out of the loop, this should improve performance.